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THE ENUMERATION OF PERMUTATIONS WITH A 
PRESCRIBED NUMBER OF “FORBIDDEN” PATTERNS 


John Noonan and Doron Zeilberger 


Abstract. We initiate a general approach for the fast enumeration of permutations with a prescribed number of 
occurrences of ‘forbidden’ patterns, that seems to indicate that the enumerating sequence is always P-recursive. 
We illustrate the method completely in terms of the patterns ‘abcVcab’ and ‘abed’. 


0. Introduction 

The only increasing permutation on {1, 2,..., n}, [1, 2,..., n], has the property that it has no decreasing 
subsequence of length 2, i.e. there are no i and j such that 1 < i < j < n, and 7r[i] > 7r[j]. Thus one measure 
of how scrambled a permutation is, is its number of inversions, inv(7r), which is the number of occurrences 
of the ‘pattern’ ba. 

This can be generalized to an arbitrary set of patterns. Given a permutation on {1, ... ,n}, we define a 
pattern as a permutation on {1, ..., r} where r < n. For r < n, we say that a permutation a G Sn has the 
pattern tt G 5^ if there exist 1 < < ^2 < • • • < ir < u such that tt = [cr(ii), cr(i 2 ),..., cr{ir)] in reduced 
form. 

The reduced form of a permutation ct on a set {ji, j 2 , • ■ • ,jn} where ji < j 2 < ■ ■ ■ < jn is the permutation 
(Ti G Sn obtained by renaming the objects of the permutation a in the obvious way, so that ji is renamed 1 
and j 2 is renamed 2 and so on. Thus the reduced form of the permutation 25734 is 14523 and the reduced 
form of 579 is 123. 

To simplify things, when discussing a particular pattern, we will use its alphabetic equivalent. Thus an 
increasing subsequence of length 3 is an abc pattern, an inversion is a ba pattern and a decreasing subsequence 
of length 4 is a deba pattern. Other patterns we will discuss include abed, bac and cab. 

The number of permutations which contain no increasing subsequence of length three (i.e. abc avoiding) 
is known to be Cn '■= Catalan numbers. It is also known [6] that given any pattern of length 

three, the number of permutations avoiding that pattern is also C„. 

Herb Wilf raised the question: For any pattern tt, what can you say about a.n.(n), the number of permu¬ 
tations on { 1 ... n} that avoid the pattern tt? 

It follows from the Robinson-Schenstead algorithm and the hook-length formula[3] that for any r, the 
number of permutations with no increasing subsequence of length r, is a certain binomial-coefficient multisum, 
from which it follows immediately [8] that it is P-recursive (holonomic) (i.e., it satisfies a linear recurrence 
with polynomial coefficients in n). 
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A natural conjecture is: For any given finite set of patterns, PAT, the sequence 


apAT{n) := 


|ct S S'n : cr has no occurrences of the given patterns| 


is P-recursive. More generally, for any such set of patterns PAT = {pati,pat 2 , ■ ■ ■ ,patr} and any specified 
sequence of integers rup, one for each pattern p G PAT, 


IpAT 


(n) := 


|cr G 5'n : cr has exactly rrii occurrences of the pattern pati, 1 < i < r| 


is P-recursive. 

In this paper, we will present a method for the ‘fast’ (polynomial time in n) enumeration of such sequences 
a{n), that seems to support our conjecture that it should always be P-recursive (holonomic) in n. 

Until later in the paper, we will consider only one pattern, as the main ideas are already present there. 
The natural object is the generating function: 


crGSn 

where p{a) denotes the number of subsequences, 0 : 1 X 2 ... Xr, present in a that reduce to the given pattern tt. 
We will later show how to derive a recursive functional equation, from which it should be possible to extract 
efficient recurrences for the coefficients Ci(n) of Fniq) for small i. As we said above, our approach seems to 
indicate that each Ci{n) is P-recursive in n. It is easy to see that it cannot be also P—recursive in i. 
Another possibility is to expand the polynomials around q = I, so that we have 


degree 

K{<1)= E UnKq-iy. 

i=0 

Here 6 o(^) is the total number of permutations, n\. bi(n) is the total number of tt = X1X2 .. .Xr patterns 
present in all the permutations of set S'„. The average number of X1X2 ... x^ patterns present in the permu¬ 
tations on {1... n} would simply be . It is easy to see [9] that it is always a polynomial in n, as are all 

n! 

the other coefficients bi{n), for each fixed i. Thus, we can compute the first few coefficients bi{n) of Ffl{q+1) 
by brute force. However to get the full F^iq) we would need the full P^(g -I- I), and the coefficients bi{n), 
while always polynomials in n, get increasingly complicated as i grows bigger. 

Instead we will answer a more general question: How many permutations on {I... n} avoid tt = X1X2 ... x^ 
and {TTi}™ where {TTi} is a set of other forbidden patterns for which one or more of the entries in the forbidden 
subsequence is specified. Among the might be the pattern a64 which simply describes an abc pattern in 
which the last entry is 4. Another might be a 36 pattern which would be a non-inversion in which the first 
entry is 3. 

Once we have such a method, we would also like to compute the number of permutations of a given size 
which avoid a given set of patterns and compute the number of permutations of a given size which have 
occurences of patterns from the given set, a prescribed number of times. We may ask how many permutations 
on {I... n} avoid both abc and cabl; how many avoid abc and have exactly I cabl; etc. 
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1 Counting Permutations with a Presribed Number of abc Patterns 

1.0 Definitions 

Definition 1.1. Given a G S'„, an abc pattern is a sequence i,j,k where 0<i<j<k<n and a(i) < 
a{j) < a{k). 

Definition 1.2. (fabc{(^)'-= the number of abc patterns of a. 

For example (/?a6c(4321) = 0, V3a6c(1234) = 4, and v3o6c(2314) = 1. 

Definition 1.3. Given cr C S'„, an aj pattern is a sequence i, k where 0 < i < k < n and cr(i) < a{k) = j. 
For example (/94(15342) = 2. 

Definition 1.4. (fijicr) := the number of aj patterns of a. 

Definition 1.5. P^'^\n,I) is the number of permutations on {1.. .n} with exactly r abc patterns and no 
aj patterns for j < I- 

We will use P(n, I) to denote P^^'^ (n, I), i.e. the number of permutations on {1... n} with no abc patterns 
and no aj patterns for j < I. 

Using the above definitions, the polynomial Fn{q) described earlier would for this example be defined as 

■= E 

So, e.g., Fi23(g) = 1 , Ffs(g) = 5 + g , PP^q) = lA + Qq + + q\ 

Definition 1.6. For a G Sm dehne: 

wt{a) := 

For example, wt(2314) = q^q 2 q\q\ = qqsqt- 

Definition 1.7. 

Pn{q2,q3,---,qn;q) ■■= E 

o-eSn 

Pn is the ‘generalized’ form of Comparing the two we see that F^‘^^{q) = P„(l, 1..., 1; g). 

1.1 The functional equation 

In order to illustrate the present method, we will treat the simplest non-trivial case, by rederiving the 
well-known formula for the number of permutations on {1.. .n} with no abc patterns by hrst obtaining a 
method to compute the number of permutations on {1.. .n} with no abc patterns and no aj patterns for 
j < I where I is any integer between 0 and n. We then merely set 1 = 0 and obtain the desired sequence. 
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In order to find explicit or recursive descriptions for the coefficients of we will establish a recursive 

functional equation for Let cr G Sn, cr(n) = i ^ 1. Let cti be the permutation on {l..i — l,i + l..n} 
obtained by removing the last entry of cr, i.e. 

O'! O') := 1 < J < « - 1- 


Then 


wt{a) = wt{ai)ql ^q^j= 2 Vji<^i)^ 


hence 

n 

wt{a) = Y[ gf H(1) 

j=l j<i j>i 

If cr(n) = I, then defining ai as above with f = 1, we have 


wt{a) = wt{ai). 

Summing over all a € SnOn the left hand side of (1) is equivalent to summing first over i, then over cti G Sn-i 
on the right. Making the necessary shift in variables to account for the fact that cri above is a permutation 
on [I, ..,i — I, i + I,n], we have 

n 

(7£Sn ^=1 o-(n)=-i C 

creSn 

n 

+ E E 

where S'^Oi is the set of permutations on {1, 2,..., z — 1, f + 1,..., n — 1, n}. 

Hence 

n 

Pniq2,q3, ■■■,qn;q) = Pn-iiq3,q4, ...,g„;g) + ^g*"^P„_i(gg 2 ,gga,• ■ •,gg*-i,g*+i,...,g„;g). (2) 

i^2 

This recurrence for is the basis for all that follows. 


<TieS„_i j<i j>i 


1.2 A RECURRENCE FOR P(n,I) 

7-1 n-7-1 

We note that from our definition of Pn, Pn(0, 0, ..., 0, 1,..., I; 0) is the number of permutations on {1 ... n} 
with no abc patterns and no aj patterns for j < I. So Hn(l, I,..., I; 0) is 0123 ( 71 ), the number of permutations 

I—l n—I—1 

on n elements with no abc patterns, and P„(0, 0,..., 0,1,..., 1; 0) = P{n, I). We first tackle the question of 
a recurrence for 0123 ( 71 ) = Pn(l, 1, • • ■, 1; 0). Using (2), we have 




i=2 


( 3 ) 
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It would be too much to hope for a nice clean recurrence on the first try, we seem to have picked up a few 
uninvited guests, namely the Pn-i{0, 0,..., 0,1,..., 1; 0). By (2) again, 

7—1 n —7 —1 7—2 n —7 —1 _ i — 2 n—i—1 


P„(0,0,...,0,l,...,l;0) = F„_i(0,0,...,0,l,...,l;0)+ ^ P„_i(0,0,..., 0,1,..., 1; 0). (4) 

i=/+i 

In terms of P{n, I), (4) becomes 


P{n, I) = P{n-l,I-l)+ P{n-l,i-l) = Y “ 1. * “ !)■ (5) 

z=74-l i—I 

The number of permutations on n elements with no abc patterns is Pn(l, 1,. •., 1; 0) = P{n, 1) = ai 23 (n). 
Unfortunately when we try to use (5) to find P{n, 1) we are required to define P{n, 0). To do so, one needs 
only look as far as (2). By (2), we have 


P(n, 1) = P(n- 1,1) + 

1=2 

Comparing this with (5) we see that we should define P{n, 0) = P{n, 1). Using this definition, (5) is valid 
for n > 1, f > 0. To complete the scheme, we need some form of initial conditions. These are readily supplied 
by the observation that the number of permutations on n elements with no abc patterns and no aj patterns 
2 < J < n is 1, namely the permutation [n, n — 1,..., 3, 2,1], so P{n, n) = 1. 


Finally, we may simplify (5) by examining 

P{n, I) — P{n, / + 1) = X]r=/ “ 1) * “ 1) “ Y^'i=i+i “ 1+ ~ 1) = “ 1) “ 1) which gives us 

{ 1, if n = / 

P(n, 1), if / = 0 (6) 

P(n, / + 1) + P{n —1,1— 1), otherwise 

Using this recurrence, we may quickly generate a large number of P{n,I). 


Table 1 

Values of P{n, I) 


n 

1=0 

1 

2 

3 

4 

5 

6 

7 

0 

1 








1 

1 

1 







2 

2 

2 

1 






3 

5 

5 

3 

1 





4 

14 

14 

9 

4 

1 




5 

42 

42 

28 

14 

5 

1 



6 

132 

132 

90 

48 

20 

6 

1 


7 

429 

429 

297 

165 

75 

27 

7 

1 

8 

1430 

1430 

1001 

572 

275 

no 

35 

8 

9 

4862 

4862 

3432 

2002 

1001 

429 

154 

44 

10 

16796 

16796 

11934 

7072 

3640 

1638 

637 

208 
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This enables us to conjecture, and immediately prove (by verifying (6) and the initial conditions), the 
closed form P{n,I) = the celebrated ballot numbers[3]. Evaluating at / = 1 yields yet another 

proof of the well known fact that the number of permutations on {1... n} with no abc patterns, 0123 {n ), 
equals Cn, the Catalan number. This proof is longer and far less elegant than the combinatorial proofs of 
[ 6 ]. Its only virtue is that it illustrates a general method. 

1.3 The number of permutations with exactly one abc pattern. 

Recently one of us [4] proved that the number of permutations on {1... n} with exactly one abc pattern 

is We will now present an alternative proof using the present method. 

The number of permutations with exactly one abc pattern is the constant term of the derivative of the 

d 

function F = 'J2a-eSn . By differentiating (2) with respect to q, we find a recurrence for '^Pn, 

d d ^ ■ d 

■^Pniq2,---,qn;q) = ( 93 , ••• ,?«;<?) + '^ql~^-^Pn-iiqq2,qq3,---,qqt-i,qt+i,---,qn;q)- (7) 

Here things begin to get a bit tricky and we will need to employ the chain rule. We will evaluate the right 
side of the above equation in terms of partial derivatives with respect to the positions that q occupies. For 
example, 


— P3(qq2 as- q) = q^ — P^ ^ + — P 3 ^ 

q^ q q^ q 

The preceding notation means: “first find the partial derivative of the function P 3 then make the necessary 
substitutions for au” Continuing we have 
d d 

-^Pn{q2,q3,---,qn-,q) = -^Pn-i(q3,q4,---,qn;q) 


^-l 9 


qk—^qqk, 2 < k<i 
qk^qk+i, i < k<n 
q-^q 


i=2 


qk^qqk, 2 < k<i 
afc^afe+i, i < k<n 
q^q 


We are now ready to tackle — P„ a ^ 0 

oq q, 1 , 2<i<n 


TrPn qk^^, 2<k<n 
dq q^O 


= -^Pn-i 2 <A:<n-l +V 

Hn ^ , n nn 


qk^O, 2<k<i dPn-i qk^O, 2<k<i 


dq qk^l, i<k<n 


- 1 +E' 


dq, i<k<n-l ' 


It is readily seen that P9l(^n, I), the number of permutations on {1.. .n} with exactly one abc pattern and 
no aj patterns for j < I can be expressed as: 


p(l) T\ — —p *-0: 2<fc</+l 


dq gfe —*■ 1 
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Furthermore, to simplify notation, let 


:= 


dPn 


dq. 


Qk ^ 0, 2 < k < I + 1 
q^O 


The actual meaning of (n, I) is immaterial, but the astute reader will see that combinatorially, Pj^'^ (n, I) 
is the number of permutations on {1... n} with no abc patterns, no ak patterns for k < I, k j, and exactly 
1 aj pattern if j < / or at least one aj pattern if j > /. The recurrence for P^^\n, I) follows from (8): 


P^^\n,I)=P^^\n-lJ-l)+ 


i=i+i 


j=i+i 


(9) 


Disregarding for a moment that we do not yet know what (n, I) is, we can state ‘initial’ conditions 
for this recurrence. First we should define P*^^^(n, 0) = P*^^^(n, 1). We can easily compute P^"^\n,n — 2). 
P^^^ (n, n —2) is the number of permutations on {1... n} with exactly 1 abc pattern and no aj patterns for j < 
n—2. If a is one such permutation then a has the form [n—2, n—1, n—3,..., n—i, n, n—i — 1, ..., 2,1], for some 
i. There are exactly n — 2 such permutations, hence P*-^^ (n, n — 2) = n — 2. Our ‘initial’ conditions (perhaps 
they should be called ‘boundary’ conditions) for this recurrence are P^^^(n,0) = P^^^(n, 1), P^^^(n,n — 2) = 
n — 2. 

To obtain a recurrence for P^^\n,I) we must return to (2) and take the partial derivative with respect 
to qj. We begin for the case when j > 3. 
d 


dq. 


■Pn{q 2 ,---,qn]q) 


-Pn-1 


dqj-i 




E i—1 ^ n 

- ^ri-1 

q^q ,=2 


<?fc ^ qqk, 2 < k <i 
qk qk+i, i<k<n-l 
q^ q 


d 




For j > 3 it follows that 


II Z + 0 - 1 ) 9 ^ ^Pn-l{qq 2 , . . . , qqj-l,qj+l, • ■ • , 9 ) ■ 

q^ q 


i-i 


Pj^\n,I) = Pjl\(n - 1,7- 1) + Y - 1,* - 1) + (j-l)x(j > 7)P(n - 1, j - 1). 


7=/+l 


where x(statement) = 1 1 ’ (®)> need to know the values of pj^^ (n, I) 

for which j < I so we have 

( 10 ) 


P,«(n,J)=P^,)(n-1,7-1). 


>(i) 


When j = 2 we have 


pW(n,7) = ^P, 


d 


dq2 

= P(n- 1,7- 1) 


1—2 n—I 

Qk ^0? 2<A;</H-1 _ P /n ^ n ^1 ^ 1 '. n\ 

qk^l, I<k<n -Pn-i(0,...,0,l,...,l,0) 
q^O 


( 11 ) 
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Using (11), the recurrence (10) simplifies to 


I) = P;^;(n -!,/-!)= - 2,1 - 2) = ■ ■ ■ = P^^> {n - j + 2,1 - j + 2) 


5 ( 1 ) 


5(1) 


( 1 )/ 



= Pin — 

J + J + !)• 





Now we can summarize oui 

■ results so 

far in 

a recurrence for P'^^\n,I) (see 

(9)). 


P^^\n 

n 

>^) = E 

P^^\n- 

i-1 

lu- 1)+ E Pi'IT'-jP- 

■ 3 ) 





L 


j=I+l 



From (12) 

it follows that 







P^^\n,I) 

-P(i)(n,/ + 1) 

= F(i)(n- 

- 1 ,/- 

1) 

n j i—1 

+ E E Pin-j,i- j) 

— 

i-1 

E 






i=I yj=I+l 


i=i+2 



= P^^\n- 

- 1 , 1 - 

1) 

n 

+ Pin- I -l,i- I - 

1) 







z^/+2 





= p(i)(n- 

- 1 , 1 - 

1) 

+ P(n-/,2). 




( 12 ) 


The last equation follows from (5). Finally we have the following recursion for P^'^\n, I): 

( P(i)(n,l), if/ = 0 

p(i)(n,/) = } n-2, if/ = n-2 

[ (n, / + 1) + P^^^ (n — 1, / — 1) + P{n — 1,2), otherwise 


Table 2 

Values of P^^'>{n,I) 


n 

1=0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

0 

0 











1 

0 

0 










2 

0 

0 

0 









3 

1 

1 

0 

0 








4 

6 

6 

2 

0 

0 







5 

27 

27 

12 

3 

0 

0 






6 

no 

no 

55 

19 

4 

0 

0 





7 

429 

429 

229 

91 

27 

5 

0 

0 




8 

1638 

1638 

912 

393 

136 

36 

6 

0 

0 



9 

6188 

6188 

3549 

1614 

612 

191 

46 

7 

0 

0 


10 

23256 

23256 

13636 

6447 

2601 

897 

257 

57 

8 

0 

0 


From this, one conjectures that P^^\n,I) is given by the expression 


gin,I) := 


2n-/-l 

n 


2n-I-l 

n+3 


(2n-2I-2 
~*"V n—I—A 


2n-2I-2 

n—I—1 


2n-2/-3 

n—I—4: 


2n-2/-3 

n-I-2 


Since it is readily verified that g{n, I) also satisfies the same recurrence and initial conditions, we have a 
rigorous proof that P^^''{n,I) = gin, I). Plug in / = 1 and we find that 0 ^ 23 ( 11 ) = di'^^ 1) = 'KnTs) 
proved in [4], 
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1.4 The number of permutations with exactly 2 abc patterns 
( 2 ) 

We now will compute 0^23 = the number of permutations in Sn containing exactly two increasing subse¬ 
quences of length 3. If 


then 


= 0123 (r) + -f 0^23 + • ■ • 

O-eSn 


^ = 20 ^ 23 ( 0 ) + 60 ^ 1^3 (n )(7 -b ... . 


dq^ 


So ai 23 (n) is half the constant term of fa ■ Similarly, by our definition of pF){ri,I) (definition 1.5), we 


•^123'. 

see that 


where 


dq' 






Qk^O, 2<k<i 
gfc^l, i<k<n—l ■ 
q^O 


We will find a recursive formula for 

From ( 8 ), we have 

52 92 

—Pn{q2,q3,---,qn;q) = -^Pn-iiqs, q4, ■ ■ ■, qn; q) 


dq 


n 


+ E 

i=2 

i-1 




dq- 


;Pn-l 


i-1 


d d 


qk^qqk,2<k<i ,Vo——P , 
qk^qk+i, * < k<n + Qq^ Qq^-^ 
q^q — 


I ^ ^ p 

/ . 9j I o a Pn—1 
^ \ dq dq, 


qk^qqk, 2 < k<i 
qk^qk+i, i < k<n 

q-^q 


J=2 

i-1 


qk^qqk, 2 < k<i 
qk^qk+ 1 , i < k<n 
q-^q 


„ d d ^ 

/ , qm a a T>i_l 


m=2 


dqm dq. 


qk^qqk, 2 < k<i 
qk^qk+i, i < k<n 
q^q 


.(13) 


d d 


from (13) it follows that 


2<k<i 

9fc^l 
<7^0 


.(2) 


92 d 


Let ^\V-s{n,I) = -^-^Pn 1 and -.(n,/) = 

(ij)^ ’ ^ dqdqj 9fc^l> i<k<n-l b,m)\ > ^ (99^ 95; 


-Pr, 


'j tyym 


qk ^0) 2<fc<Z Tlipn 

( 7 fe^l, i<A:<n—1 ’ 

9^0 


$(2)(„,/)=^ 


2=/ 


d>(2)(n-!,*-!)+ ^ 2<i>g^.)(n-l,z-l)+ ^ (n - 1 , * - 1) 

j=/+l \ m=J+l y 


Subtracting successive terms: 

$(2) (n, /) - $(2) (n, / -b 1) = $(2) (n - 1, / - 1) 


IL 

z =/+2 

n i—1 

E E 


,( 2 ) 


,( 2 ) 


( 14 ) 
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We note now that to compute we must also compute and <i)|^/), but we do not 

need to compute these for all n,I,j,m. We need only compute them for j, m < I. We use (8) to obtain a 
recursive formula for /), j < I. 


-^-^Pn[q 2 , qs, ■ ■ ■ , Qn] q) 


J-1 


= E<^r 


2=2 


2-1 

'y ^ q-m 

m—2 


d d 
dqni ^qj—l 


Pn-1 


qk^qqk, 2 < k<i , ^ Q p 
qk^qk+i, i < k<n dq dqi-i " 

q^q 


qk^qqk, 2 < k<i 
qk^qk+ 1 , i < k<n 

q^q 


n 

+q E 

i=j+l 


i-1 

E 

m—2 


d d 

dqra dqj 


Pn-l 


qk- 

-^qqk, 2 < k<i 

d 

Ap , 

qk- 

^qqk, 2 < k<i 

qk- 

-^qk+i, i < k<n 


uQj 

qk- 

^qk+i, i < k<n 

q- 

^q 


q- 

^q 


+ (i - P)q] ^ 


A d 

qk- 

-^qqk, 2 < k<j 

+ Ep , 

qk^qqk, 2 < k<j 

qk- 

-^qk+i, j < k<n 

oq 

gfe^gfc+i, j < k<n 


q- 

^q 

q^q 


n 

p E 

i=J+l 


9g- 


■Pn-1 


qk^qqk, 2 < A:<t , ^ ^ p _ 

qk^qk+i, i < k<n dq dqi-i ” ^ 


qk qk+l 

q^ q 


From this, we have 


$ 


( 2 ) 

(Ij) 


(n,/) 


E”=i+i ^(2) (n -!,*-!)+ P^^Hn -1,1- 1), if j = 2 
Er=7+i P^in -!,*-!) + - 1, / - 1), if 2 < j < J 


(15) 


where Plp{ri,I) = 


qk^qqk, 2 < fc</ 
gfc^gfc+ 1 , I < k<n ■ 
q^q 


This recurrence can be further simplified using (12) to 


n—j-\-k 


^(ij) =-P^^H»^-j + l.-^-J + l) + E E P{n-j,i-k)j , lor j < I. 

k—2 \i—I~\-k—j-\-l 


(16) 


( 2) 

We may do the same for ^ (n, /) and obtain 


$ 


( 2 ) 

(m,j) 


0 , 

2 P(n- 1 ,/- 1), 

.(2) 


if J = m = 2 
if j = TO = 3 < / 




P«,)(n-1,/-1), 

I <h[l,,^_i)(n-l,/-l), 


which collapses down to the 3 cases 


$ 


( 2 ) 


, (n, d) = < ‘2P{n — j + l,I — j + 1), 


if j > TO = 2 
if j > TO, m < I 


if j = TO = 2 

if j = TO < / 


P, 


( 1 ) 


-m+i) {n - m + I, I - m + 1) , if j > to, to < 


(17) 
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One might think that we have overlooked a few cases here, for what if j > m > / or j = m > /? Examining 
(14) it is apparent that to compute we need only compute j) and when j, m < I. We 

now use (5),(12),(16) and (17) to simplify (14): 

$0) (n, I) = (n - 1, / - 1) + (n, J + 1) 

+ 2P^^\n- 1,2) + 2IP{n- I,S) + 2P{n-i- l,2) + x(/ > l)2P(n - / + 1,3) 

We may now make the substitution P^'^\n,I) = /). Our recurrence for P^^'>{n,I) can be stated as 

P^'^^{n,l), if 7 = 0 

n — 3, if J = n — 2 

P(2) („ _ / _ 1 ) + p(2) 7 + 1) + P(i) {n -1,2) 

+P(n - t - 1,2) + IP{n - 7,3) + x(7 > l)P(n -7+1,3), if 0 < 7 < n - 2, n > 3 


P^^\n,I) = 


Table 3 

Values of P^'^^ (n, I) 


n 

1=0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

2 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

3 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

4 

3 

3 

1 

0 

0 

0 

0 

0 

0 

0 

0 

5 

24 

24 

12 

2 

0 

0 

0 

0 

0 

0 

0 

6 

133 

133 

74 

23 

3 

0 

0 

0 

0 

0 

0 

7 

635 

635 

371 

141 

36 

4 

0 

0 

0 

0 

0 

8 

2807 

2807 

1688 

709 

227 

51 

5 

0 

0 

0 

0 

9 

11864 

11864 

7276 

3248 

1168 

334 

68 

6 

0 

0 

0 

10 

48756 

48756 

30340 

14121 

5459 

1771 

464 

87 

7 

0 

0 


Using ordi of [10] or gfun[5] we conjecture that 

( 2 ) , , 59n2 + 117n+100/ 2n \ 

2 „( 2 „-!)(„ + 5 ) 

It is very likely that one should be able to conjecture an explicit expression for P^^\n, I), which would be 
routine to prove, and from which the above conjecture would follow. 

2 The Method 

We may now outline the method described in this paper. 

To determine the number of permutations on {l...n} having exactly r occurrences of the pattern 


XlX2 ■■ - Xk 
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1. Determine the best way to obtain a recurrence for this pattern. 

There are basically four ways to do this. 

a) by removing the last entry of the permutation, as in the abc example. 

b) by removing the first entry of the permutation, which is what we will do in the next example. 

c) by removing n from the permutation. 

d) by removing 1 from the permutation. 

2. Identify the other parameters needed in order to describe the recurrence. 

In our first example, we found a recurrence for the number of permutations with a given number of 
abc patterns by looking at what happened to a permutation from which we removed its last object. As a 
consequence, we were forced to consider the number of abc patterns present in each permutation, but also 
the number of aj patterns present. Note that we could arrive at this requirement by noting the result of 
removing the last object from the pattern abc. abc becomes ab. Since only some choices of b result in a true 
abc pattern for a given c, we must be specific, and count the number of aj patterns for every j. We count 
the number of aj by using the parameter qj in our weight function. 

3. Define P„ = J2a£Sr, 

Here wt{a) = H where Pjia) is the number of times the pattern associated with qj can be 

found in a and ip{<j) is the number of xiX2 .. .Xr in ct . 

4. Determine the functional equation. 

Using the recurrence described in 1, determine a recurrence in n for P„. 

5. Take the derivative of the functional equation with respect to q. 

6. Let q = 0, and qj = 1 for all other parameters of Pn. 

For all intents and purposes, you are now done, for you have obtained a (perhaps complicated) recurrence 
for the number of permutations on {1.. .n} containing exactly k xiX2 .. .x^’s. This recurrence probably 
involves other terms, but each of these have their own recurrences which can be determined from the 
functional equation. After all is said and done, you frequently are able to simplify this recurrence, eliminating 
many of these unwanted terms, but for now, you have achieved your goal. 

3 The forbidden pattern cab 

3.1 Definitions 

As with each of the examples we examine, the definitions for P, P'^^\ (pj and wt should be taken as being 
local to the problem at hand. 

Definition 3.1. Given a € Sn, a cab pattern is a sequence i,j,k where l<i<j<k<n and a{j) < 
a{k) < a{i). 
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Definition 3.2. For a S 5'„, let ipcabio') be the number of cab patterns of a. 

The number of permutations on {1... n} having no cab patterns will be the constant term of the polynomial 
F = . As we saw earlier, we may add any number of parameters to this polynomial, as long 

as we know what to do with them to obtain the constant term. In this case, we will use the parameters 
< 72 , as we did with abc. 

Definition 3.3. Given a € Sn, 

n 

Wt{a) 

f=2 


3.2 No ca&’s 

Let (j{n) = i. A functional equation results from examining ai{k) = a{k), 1 < k < n, 

wt{a) = 

j>i 


Summing over all a G Sn we have 

n 

Pniq,q2,q3,---,qn) 




Y[qj I Pn-i{q,qq2,---,qqz-i,qz+i,---,qn) 

j>i 


Y[qjj Pn-i{q,q3 ,---,qn)- 
(18) 

7—1 n—7—1 

Not surprisingly, if we let P{n, I) = P„(0; 0,0,..., 0,1,..., 1), then we obtain the following recurrence for 
P(n,/), 

{ 1 , if n = / 

P{n, 1) , if J = 0 

P{n, I + 1) + P{n — 1, / — 1), otherwise 

Which is identical to (6). So we have reproved that the number of permutations on {1.. .n} with no abc 
subsequences is equal to the number with no cab subsequences. 

3.3 Permutations with one cab 

Though the number of permutations with no abPs is equal to the number with no ca6’s, we will find 
that this is not the case when we examinee the number of permutations with one cab. we follow the same 


d 

procedure outlined for the abc case, with the goal of finding — 

oq 

of both sides of (18) with respect to q, we have 


dq^ 


q ^ 0 . Taking derivatives 

^ 1, 1 < j < n 



i-1 

^ P 

rx n—1 

OQj 

qk- 

qk- 

-^qqk, 2 < k<i , n 

i < k<n dq 

qk^qqk, 2 < k<i 
qk^qk+i, i < k<n 

\j>i ) 

_J=2 


q- 

yq 

q^q 


TT ^ 7-» 


q-^ q 

qk qk+i, I <k <n-l 
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We do the same for the derivative with respect to Qj: 


A 

dqj 


1-1 

( \ 

Pn=J2 

Qk 1 Pfi—l 

i^2 

k>i I 

\k^j / 


Qk^qqk, 2 < k<i 
qk^qk+i, i < k<n 
q-^q 



qk^qqk, 2 < k<i 
qk^qk+i, i < k<n 
q^q 





k>i 


qk^qqk, 2 < k<i 
qk^qk+ 1 , i < k<n ' 
q-^q 


Here we let = ■§^Pn 

following recursions: 


q^Q 

% ^ 0, k <I 
qk^l, k > I 


and Pj-^hn,!) = 


^ ^ 1 , ^ T ■ We obtain the 

qk —^0, k < I 

qk k > I 


( 0 


PSl{n.I) = { 


Pmi«- 1). 


a n — I<0 or j<I 

U) if / = 0 

P(n-1,/-1), ifj=/ 

I ECi Pi'iT' - 1, ^ - 1), if J > ^ 


I, 


if / = n — 2 

if / = 0 
otherwise 


p(P{n,I) = } P^^\n,l), 

[ Eti [EA+1 - 1, * - 1) + P^^Kn - 1, z - 1) 

We combine these two recurrences and localize to obtain 

I, if / = n - 2 

p(^^{n,I) = <( 1), if 7 = 0 

I + 1) + pP^{n — 1, / — 1) + P{n — 2,1), otherwise 


Table 4 

Values of P^^'>{n,I) 


n 

1=0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

0 

0 











1 

0 

0 










2 

0 

0 

0 









3 

1 

1 

0 

0 








4 

5 

5 

2 

0 

0 







5 

21 

21 

11 

3 

0 

0 






6 

84 

84 

49 

19 

4 

0 

0 





7 

330 

330 

204 

92 

29 

5 

0 

0 




8 

1287 

1287 

825 

405 

153 

41 

6 

0 

0 



9 

5005 

5005 

3289 

1705 

715 

235 

55 

7 

0 

0 


10 

19448 

19448 

13013 

7007 

3146 

1166 

341 

71 

8 

0 

0 


Using ordi[10] or gfun[5] we conjecture that a 


( 1 ) 

312 


(n), the number of permutations on {1, 2 ,..., n} con¬ 


taining exactly one cab subsequence, = 
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4 Counting abed’s 

Lest the reader think that the methods outlined in this paper will only help us gain information about 
permutations avoiding forbidden patterns of length three, here we examine the forbidden pattern abed. 


4.1 Definitions 

Definition 4.1. An abed pattern of a permutation u is a sequence l<i<j<k<l<n with a{i) < 
a{j) < a{k) < a{l). 

Definition 4.2. ipabcdio') = the number of abed patterns which can be found in a. 

As before, we have 

Definition 4.3. (paj{<j) = the number of aj patterns of cr. 

Here we must also define the following. 

Definition 4.4. An abj pattern is a sequence 1 < ii < i 2 < is < n with cr(ii) < a{i 2 ) < o-{is) = j. 
Definition 4.5. (f abj (o') = the number of abj patterns of a. 

Definition 4.6. Let a € Sn then 

n n 

wt{a) = 

j=3 j=2 

The number of permutations with no abed 


Let (Ti(j) = cr(j), 1 < j < n — 1. If cr(n) = i then 




(19) 


Let Pniq;q 3 , ■■■,qn;^ 2 ,---,^n) = wt{a). Using (19), we have 

n 

Pn{q] 93) ■ • ■ , <Zn; ^2, • ■ • , ^n) = ^ Pn — l{qi 993, • ■ • , 99i-l) 9i+l, • ■ • , 9n, 9i?2, • ■ • , qi^i—1, Ci+1) ••■,?«) 

i=3 

T ^2Pn — l{qi 94, ■ ■ ■ ) 9n! ^3) ■ ■ • ) ^n) + Pn — liqi 94, ■ ■ ■ , 9n j ^3, ■ ■ • , ^n) ■ (20) 

The number of permutations on { 1 ... n} with no abed patterns is 

P„(0; 1, ..., 1; 1,..., 1) = ^ P„_i(0; 0,..., 0,1,..., 1; 1,..., 1) + 2P„_i(0; 1,..., 1; 1,..., 1). 

We may use (20) twice more to find 

7i-2 n-h-l 


h 


7i—3 n —7i —1 i — 2 n — i—1 


P„(l; 0,..., 0,1,..., 1; 1,..., 1) = ^ P„_i(0; 0,..., 0,1,..., 1; 0,..., 0,1,. .TTl) 


i=3 

„ i-3 n-i-1 

+ ^ P„_i(0;0r?y0,ir?^;!,...,!) 

i—Ii + 1 

7i—3 n—7i —1 

+ 2P„_i(0;C^, 1, • • •, 1) 
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and 


I\—2 n — — l /2 — 1 n—l 2 — l ^ 1 —3 n—Ii — 1 i — 2 n—i — 1 

*—/2+1 

^ i —3 n — i—1 / 2 —2 n —/2 —1 

+ ^ P„_i(0;0r^,C^;C^,C^) 

/i—3 n—Ii—1 / 2 —2 n —/2 —1 

+ P„_1 (0; C^) • 


( 21 ) 


Ii—2 n—Ii — 1 /2 —1 n —/2 —1 

Let P(n,/i,/ 2 ) = Fri(l; 0,..., 0,1,..., 1; 0,..., 0,1,..., 1) for 0 < {/i,/ 2 } < n. Observe that P{n,n,n) 
is the number of permutations on with no abed pattern, no abc pattern (for any c) and no aj 

patterns for any j. There is only one such permutation, namely [n, n — 1,..., 2,1], thus P{n, n,n) = 1 for 
all n. Furthermore, it is clear from (21) that when Ji < I 2 , P(n,/i,/ 2 ) = P(n, 12 ,^ 2 ). We may define 
P{n,li,0) = P{n,Ii,l) and P{n,0,I2) = P(n, l,/2) for all values of n, Ii and l 2 - Using this notation, we 
have 


U(n,l,/ 2 ), 


if Ji = 0 

P(n,/i, 1), 


II 

0 

P{n,h,l2), 


if h < h 

1, 


II 

h 

n 



^ P(n-l,/i-l,z-l)+ ^ P(n-l,z-l,J 2 ) 
i=/ 2 +i i=/i+i > Otherwise 


+ P(n-l,/i-l,/ 2 -l) 


Table 5 

Values of P(n, /, 1) 


n 

1=0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

0 

0 











1 

1 

1 










2 

2 

2 

2 









3 

6 

6 

6 

5 








4 

23 

23 

23 

20 

14 







5 

103 

103 

103 

92 

70 

42 






6 

513 

513 

513 

466 

372 

252 

132 





7 

2761 

2761 

2761 

2536 

2086 

1509 

924 

429 




8 

15767 

15767 

15767 

14594 

12248 

9227 

6127 

3432 

1430 



9 

94359 

94359 

94359 

87830 

74772 

57894 

40403 

24882 

12870 

4862 


10 

586590 

586590 

586590 

548325 

471795 

372565 

268909 

175474 

101036 

48620 

16796 
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Table 6 

Values of P{n, 1, 1) 


n 

1=0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

0 

1 











1 

1 

1 










2 

2 

2 

1 









3 

6 

6 

3 

1 








4 

23 

23 

12 

4 

1 







5 

103 

103 

56 

20 

5 

1 






6 

513 

513 

288 

no 

30 

6 

1 





7 

2761 

2761 

1588 

640 

190 

42 

7 

1 




8 

15767 

15767 

9238 

3882 

1235 

301 

56 

8 

1 



9 

94359 

94359 

56094 

24358 

8187 

2163 

448 

72 

9 

1 


10 

586590 

586590 

352795 

157265 

55235 

15575 

3528 

636 

90 

10 

1 


Using ordi[10], we conjecture that 0 ^ 234 (n) = P(n, 1,1) satisfies the following recurrence with 
^1234(0) = 1 ^1234(1) = 1- 


JO) 


5 Counting permutations avoiding more than one forbidden pattern 


The method outlined in this paper can be used to find recurrences for the number of permutations avoiding 
2 or more forbidden patterns. The method is essentially the same, though you have to keep track of more 
parameters, and recurrences can be more complicated. In the example that follows, we seek the number 
of permutations on {1.. .n} avoiding both abc and bac. Let ipbac{o') be the number of bac patterns of the 
permutation cr. Let (pja{o') be the number of ja patterns of a (that is the number of inversions ‘caused’ by 
j). Let 

n 

Wt{a) = JJ- ^ 

1 = 2 

It may seem a bit wasteful to spend time defining both ipaj{o') and (pja{cr) when it is clear that (pajicr) + 
= n — 1 but we will see that this ‘complication’ in addition to the introduction of the parameters 
will pay off in the end. 


Pn{q;^;q 2 ,---,qn;^ 2 ,---,^n) = wt{a) ^ wt{a) 

aes„ i=l 

“ ^ ^ Qi ^ -^n —1 (q? QQ 2 i ■ ■ ■ 7 QQi—lf Qi+ 1 1 • ■ • iQn'f ■ ■ ■ 7 17 7 • ■ ■ 7 '?n) 

Pn—liQ] ■ ■ ■ 7 ^37 ■ • ■ 7 ^n)- 


\j>i 
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7—1 n—I 7—1 n—I 

Let P{n, I) = P„(0; 0; 0,..., 0,1,..., 1; 0,..., 0, Note that from our definition, P(n, /) = 0 when 

/ > 1. Indeed, P{n, I) is the number of permutations on {1... n} containing no abc, no bac, no aj for j < I 
and no ja for j < I. If there was a permutation for which this was true for an 1^1 then we merely examinee 
the positions of the object 1 and 2 in the permutation. Let a(i) = 1 and a{j) =2. If i < j then a has an a2 
pattern. If j < z then a has a 2a pattern. Thus, cr does not meet the requirements and so P(n, I) = 0 for 
/ > 1. We have the recurrence 


P{n,I) 


P(n,l), 

0 , 

1 , 


if / = 0 
if / > I 
if n = / = I 
I), otherwise 


This recurrence reduces to 


P(n,l) 


1, if n = I 

2P(n-I,l), ifn>r 


So we have reproved the well know result that the number of permutations on {1... n} with no abc and no 
bac is 2”. 


Conclusion 

We saw in the above examples that in order to compute the quantity of interest, a„, say, we naturally 
introduced extra parameters Ii, / 2 ,..., and a new quantity F{n, ■ ■ ■) such that a„ was the special case 
F(n, 1,1,...), in which all the extra parameters Ii, I 2 , ■ ■ ■, are set to 1. Since the system of linear recur¬ 
rence equations always seems to have constant coefficients, the generating function in all the corresponding 
continuous variables 


F{z;xi,X 2 ,...) := ^ F{n, h, h, ■ ■ ■)z'^x{^xiF .. 

nji , 72 ,... 

should be holonomic (multi-D-finite) in all its variables (because of the nonstandard boundary conditions, it 
is not always a rational function). It follows from the holonomic theory [9] that any coefficient with respect 
to xi, 012 , • ■ •, in particular that of x\x 2 ■ ■ ■, is still holonomic (D-finite, i.e. satisfies a linear differential 
equation with polynomial coefficients), hence /„ := F{n, 1,1,...) is P—recursive. 

The method described here works for many patterns and sets of patterns, but it does not seem to work 
for all patterns. The authors were unable to find a suitable set of parameters (see the method, step 2) to 
apply this method to the pattern adcb. 


Appendix 

A maple package which confirms and illustrates many results from this paper is available and can 
be obtained using your favorite world wide web browser at http://www.math.temple.edu/~noonan or 
http://www.math.temple.edu/~zeilberg. 



THE ENUMERATION OF PERMUTATIONS WITH A PRESCRIBED NUMBER OF “FORBIDDEN” PATTERNS 19 


References 

1. Kimmo Erikson and Svante Linusson, The size of Fulton’s essential set, The Electronic Journal of Combinatorics 1, R6 
(1995), 18pp. 

2. V.E. Hoggatt, Jr., and Margorie Bicknell, Catalan and related sequences arising from inverses of Pascal’s triangle matrices, 

The Fibonacci Quarterly 14 (1976), 395-404. 

3. Donald Knuth, The Art of Computer Programming , vol 3, Sorting and Searching, Addison-Wesley, 1973. 

4. John Noonan, The number of permutations containing exactly one increasing subsequence of length three, Discrete Math¬ 
ematics to appear. 

5. B.Salvy and P. Zimmermann, Gfun: a Maple package for the manipulation of generating and holonomic functions in one 
variable, ACM Trans. Math. Soft. 20 (1994). 

6. Rodica Simion and Frank W. Schmidt, Restricted Permutations, European Journal of Combinatorics 6 (1985), 383-406. 

7. Julian West, Permutations with forbidden subsequences and stack sortable permutations, PhD thesis, MIT (1990). 

8. Doron Zeilberger, A Holonomic systems approach to special functions identities, J. of Computational and Applied Math. 

32 (1990), 321-368. 

9. Doron Zeilberger, The Joy of Brute Force, available via world wide web browser at http://www.math, temple.edu/ "zeil- 
berg/papersl.html. 

10. Doron Zeilberger, SCHUTZENBERGER, a package for maple, available via world wide web browser at http://www.math, temple. edu/ 
"zeilberg/programs.html. 

Department of Mathematics, Temple University, Philadelphia, PA 19122 

E-mail address: noonainOmath. temple, edu, zeilberg@math.temple.edu, WWW homepage: http://www.math.temple.edu/''nooncui, 
http: //www. math. temple. edu/''zeilberg 



